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ABSTRACT 

A study examined whether the state and federal 
performance standards f ^r Job Training Partnership Act (JTPA) 
programs have influenced Service Delivery Area (SDA) and service 
provider decisions about program design and implementation practices 
in ways that have affected the clients served, the types of services 
provided, and the costs of the services. Only a few SDAs appeared to 
give priority to their performance goals at the expense of other 
program objectives. It was only in those SDAs without well-defined 
client or service goals rhat performance standards had notable 
unintended effects. A great deal of variation was found in the way 
states have chosen to use their authority to establish JTPA 
performance standard policies and to set priorities for the types of 
clients served, states that established target groups for JTPA 
programs did influence SDAs to enroll more hard-to-serve clients. 
There was no evidence to suggest that federal standards for the 
entered employment rate and wage rate for adults generally had 
unintended effects on clients or services. On the other hand, federal 
cost standards did have a number of unintended effects and were the 
least comparably measured of all federal performance measures. 
(MN) 
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September 1988 

TO THE PRESIDENT AND THE CONGRESS OF THE UNITED STATES 

The National Commission for Employment Policy (NCEP) is pleased to present to you this 
Executive Summary of our major research study entitled, Evaluation of the Effect ofJTPA Per- 
formance Standards on Clients, Services, and Cost. This is the first national evaluation of a 
cornerstone of the Job Training Partnership Act (JTPA) — the implementation of clear, 
measurable program goals that exemplify "the return on investment" strategy upon which the en- 
tire program is built In fact. Congress itself believed that performance standar(k were of such 
importance that it mandated an evaluation of their impact in the enabling legislation. Section 
106(f)(2) of the Act directs the Commission to "evaluate the impact of such standards (intended 
or otherwise) on the choice of who is served, what services are provided, and the cost of such 
service in service delivery areas." 

I believe this report is one of the major contributions of my tenure as Chairman of the Com- 
mission. Contractors working on the study, SRI International and Berkeley Planning Associates, 
are to be commended for their superb job of gathering and analyzing the diverse and compli- 
cated data. As one of our reviewers notes "this is a tremendously valuable piece of work, break- 
ing new ground in national studies." I am confident you will agree that the findings detailed are 
very useful, having implications that go well beyond the performance management system and 
beyond JTPA. Many of the findings can be helpful when enacting legislations for other human 
resource programs. 

The Commission believes that programmatic performance standaids have been a successful 
strategy to focus the management of JTPA toward the goals of the legisladon— finding produc- 
tive employment for disadvantaged people. This not to say that no adjustments are needed to im- 
prove the performance management system; however, we feel that, overall, the performance 
management system is having its intended effect of guiding JTPA resources toward meaningful 
investment in the Nation's most precious resource — ^its human capital. 

At this time, I am transmitting only the Executive Summary to you. It contains an overview 
of the study, major findings, and so-^e of the specific findings. In addition, the Commission is 
pleased to have the opportunity to present you both general and specific recommendations that 
wc feel would make the performance management system even more valuable to the Nation's 
largest job training system. Should you, or your staff, be interested in the full report, I would be 
happy to forward it tc you. 



Sincerely, 




Gertrude C. McDoupM 
Chairman 
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PREFACE 



This Executive Summary is a departure from the traditional abstract of research report 
published by the National Commission for Employment Policy. This extended Executive Sum- 
mary is provided for the reader who may not have the time or inclination to review the entire 
report wi4h its appendices. 

The Executive Summary is divided into six secHons. The first section provides an overview 
of the role of performance standards in JTPA. Next, the paper presents the objectives of the 
study. Major findings are summarized in the tliird section with detailed findings presented in the 
fourth part of the summary. The fifth section includes the implications of the findings from the 
perspective of the authors of the report. Finally, the recommendations of the Commission for 
changes to the performance management system are presented. 

The Commission hopes that the Executive Sunraiary provides a flavor of the valuable infor- 
mation contained in the complete report. Many individuals interested in JTPA and the effect of 
performance standards will also want to read the full report, which contains extensive informa- 
tion from the case studies and the quantifiable factors shown to influence who is ser\"ed in 
JTPA, the type of services received, and the cost of such services. The full report and the appen- 
dices are available free of charge from the Conmiission. 
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Executive Summary 



Role of Performance Standards 
in the JTPA Title II-A Program 

Performance standards are an integral 
part of the Job Training Partnership Act 
(JTPA) programs operated under Title II-A. 
The standards implemented during the first 5 
years of program operation [Transition Year 
1984 through Program Year 1987(PY 87)] 
were intended to further a variety of Federal 
program priorities. These included 

• To hold service delivery areas 
(SDAs) accountable for program 
outcomes, 

• To encourage the achievement of 
quality employment outcomes, 

• To encourage the achievement of cost- 
effective outcomes, 

• To create an incentive for effective 
management by local program ad- 
ministrators, and 

• To foster acceptance of the program 
by the business community. 

The focus on accountability and the con- 



cern with outcome measures — rather than 
process measures — were new to employment 
and training programs. The performance 
standards grew out of the legislative goal of 
"measuring the return on the JTPA program 
investment" in terms of increases in the 
employment and earnings of economically 
disadvantaged individuals and reductions in 
welfare dependency. Although not intended 
to be an accurate measure of net program im- 
pacts, the performance standards are u;ed to 
indicate the extent to which individual SDAs 
are managing their resources in order to 
achieve important program outcomes. 

Starting in PY 88, several additional per- 
formance standards have been implemented, 
including measures of the participants' status 
3 months after leaving the JTPA program. 
The new post-program standards are intended 
to provide greater emphasis on the objective 
of improving job retention and longer term 
employability of participants than do the cur- 
rent measures, which measure the immediate 
employment status of participants on leaving 
the program. 



ERLC 



1 

11 



The Federal Role in 

Performance Standards 

Although the performance standards are 
one expression of Federal goals for the JTPA 
programs, the performance-standards system 
has been carefully designed to minimize the 
day-to-day Federal role in local program 
direction and admini <tration. The JTPA sys- 
tem is highly decentralized, enabling local 
Private Industry Councils (PICs) and SDAs 
to design programs relevant to the needs and 
employment opportunities in their local 
areas. In particular, the local agencies are 
given wide discretion in the types of program 
services provided and the types of clients 
served. 

The Federal role in influencing the 
recruitment and selection of JTPA clients is 
limited to several requirements in the legisla- 
tion: (1) at least 90% of the enroUees be 
economically disadvantaged, (2) recipients 
of Aid to Families with Dependent Children 
(AFDC) and high school dropouts receive 
an equitable share of 'TPA services, and (3) 
40% of all JTPA service dollars be expended 
for services to youth. 

The Federal role in determining the types 
of program services provided is even more 
limited, allowing SDAs substantial discretion 
in designing their JTPA service mix. Al- 
though public service employment is not an 
allowab'^ service and expenditures on sup- 
portive services are restricted (limiting the 
ability of SDAs to offer stipends to class- 
room training participants), a broad range of 
services is permitted by the JTPA legislation 
and FederpJ program regulations. 

Consistent whh the limited Federal role 
in designing the JTPA program, the intention 
of Federal performance-standards policies is 
to foster accountability and cost-effective- 
ness without undue influence on SDA design 
decisb'i? The Federal performance-stand- 
'ds policies include the choice of measures, 

numerical level at which the standards 
and the provision of optional models 



for adjusting standards. 

Careful effort went into choosing Federal 
performance measures that would not con- 
strain SDAs in their decisions about whom to 
serve and wiiat serN'ices to offer. The numeri- 
cal level of most Federal standards is set at a 
"minimally acceptable** level so that SDAs 
could generally meet their standards without 
distorting their program designs. The adjust- 
ment models developed for the Federal stand- 
ards are intended to ensure the "neutrality** of 
the standards with respect to decisions about 
client targeting and to "hold harmless** SDAs 
that choose to serve more difficult clients 
through downward adjiYStments to the 
numerical levels of their standards. 

The State Ziole in 

Performance Stand'ards 

States can play a very visible and impor- 
tant role in the JlPA program as a whole and 
in the performance-standards system in par- 
ticular. ;cifically. States are responsible 
for setting the levels of standards for SDAs. 
States decide whether or not to use the 
Department of Labor*s (DOL) adjustment 
model to set SDA standards and make adjust- 
ments beyond the model to those standards. 
States may also develop and implement addi- 
tional performance standards to further State 
program goals and priorities and may decide 
how to weight the different State and Federal 
standards in making incentive awards. Statei 
also determine what portion of the 6% funds 
will be used for technical assistance to 
SDAs, what porL^n will be used for incen- 
tive awards to SDAs that exceed the stand- 
ards, and what portion will be used for 
incentives to serve the hard to serve. States 
determine whether the incentive awards will 
encourage perfomiance that just exceeds the 
standard level or whether there will be a 
strong financial incentive for SDAs to per- 
form at the highest level possible. Fin?lly, 
Staics determine policies to sanction SDAs 
that fail to meet their standards. 

Although the Federal performance-stand- 
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aids system was designed to encourage SDA 
discretion about who to serve and what ser- 
vices to offer, State governments are en- 
couraged to use the performance-standards 
system to further State client and service 
priorities Some States have welcomed the 
opportunity to take a leadership role in these 
areas, while other States have decided to play 
a less active role and defer to the preferences 
of local SDAs. 

The SDA and PIC Role in 

Performance Standards 

SDAs, with guidance from PICs, have the 
responsibility for actually implementing 
JTPA programs and for making major 
program-design decisions about service and 
client mix. In making these decisions, SDAs 
trj' to balance the influence of local goals, 
their assessment of local conditions, the per- 
formance-standards incentives offered by the 
State, and the State and Federal program 
regulations. SDAs are also motivated to per- 
form well on the performance standards for a 
number of reasons, including the desire to (i) 
run cost-effective programs that achieve high- 
quality outcomes, (2) appear to be account- 
able to local elected officials and private 
sector representatives on the PIC, (3) avoid 
reorganization as a consequence of failing to 
perform at acceptable levels for 2 years in 
succession, and (4) receive incentive awards. 

SDA and PICs may respond to perfor- 
mance-standards incentives in three key areas 
of program design: (1) decisions about the 
types of clients to serve and procedures to 
recruit and enroll clients; (2) decisions about 
die types of services to provide and the 
length of those services; and (3) decisions 
about how to provide services, including thw 
choices of service providers, type of contract, 
and contract terms. 

Objectives of the Study and 
Key Policy Questions 

The National Commission for Employ- 
ment Policy (NCEP) has funded a com- 



prehensive evaluation of several important is- 
sues, in keeping with its legislative mandate 
to advise the Secretary of Labor on the 
development of the JTPA performance stand- 
ards and to evaluate the impact of standards 
on the choice of who is served, what services 
are provided, and the cost of such services 
at ihe local SDA level (Section 106 (f)). The 
overall goal of tho study is to assess whetiier 
the performance standards have influenced 
SDA and service-provider decisions about 
program design and implementation practices 
in ways diat have affected the clients served, 
the types of services provided, and the costs 
of the services. 

The study is not a detailed assessment 
of whether the performance standards have 
resulted in an increase in JTPA program 
productivity and cost-effectiveness. Rather, 
the study's purpose is to test whether the in- 
centives created by the performance-stand- 
ards system have caused SDAs to emphasize 
measured performance objectives at the ex- 
pense of unmeasured or unrewarded objec- 
tives that are also valued. In particular, the 
goal is to determine whether emphasis on the 
performance standards as a vehicle for 
program management has led SDAs to avoid 
serving more difficult clients or to reduce the 
intensity of services offered, which in turn 
may have affected the types of participants 
who could benefit from the program. 

The evaluation was intended both to 
determine the effects of performance-stand- 
ards policies and to investigate how those ef- 
fects come about. Thus, the evaluation was 
designed to meet several goals. First, the 
study was designed to assess the effect of the 
overall Federal performance-standards sys- 
tem on SDA and service -provider behavior. 
Our examination of the Federal performance- 
standards system distinguished among the ef- 
fects of (1) the kinds of performance 
measures selected, (2) the level at which per- 
formance expectations were set, and (3) the 
adequacy of the adjustment model in holding 
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SDAs "harmless" for their client and service 
choices. 

Second, the study was designed to assess 
how various State performance-standards 
policies and procedures have affected SDA 
and service-provider behavior and how these 
effects have influenced the clients served, the 
services provided, and the costs of the ser- 
vices. This evaluation distinguished the ef- 
fects of several aspects of State policies, 
including (1) choices about procedures to ad- 
just standards, (2) policies for serving hard- 
to-serve clients, and (3) policies for awarding 
incentives for goodperforaiance and 
sanctioning for poor performance payments. 

When we found evidence that perfor- 
mance standards did appear to be influencing 
the clients, services, or costs, the study was 
designed to explore how and why those in- 
fluences occurred and whether the influences 
were intended (for example, causing SDAs to 
serve hard-to-serve clients and to offer ser- 
vices that will make a difference in the long 
run) or unintended (for example, causing 
SDAs tr ve fewer hard-to-serve clients or 
to offer less intensive services). 

Finally, the study was designed to 
generate suggestions for refining the perfor- 
mance-standards system so that my un- 
desirable impacts of performance standards 
can be prevented in the future. 

Major Findings 

Local Response to 

Performance Standards 

The JTPA legislation gives authority to 
SDAs to design programs to meet local 
needs. Consistent with this intent, this evalua- 
tion found dramatic differences among SDAs 
in the design and operation of JTPA 
programs, and many of th^se differences af- 
fected clients, services, and costs. Although 
performance-standards policies did affect 
some SDA design decisions, many elements 
of local JTPA design reflected local condi- 
tions and constraints beyond the control of 



the SDA. The local unemployment rate in 
particular had a strong influence on the type 
of clients in JTPA programs. In tight labor 
markets, most of the clients interested in 
JTPA programs were those who had little 
work experience or major barriers to employ- 
ment, often requiring basic skills remedia- 
tion. In contiast, in areas with high 
unemployment, individuals seeking JTPA 
services generally had rcore job skills and re- 
quired help in retraining for new industries or 
occupations. The local availability of service 
providers and of altemative services also af- 
fected the types of services that were offered 
through JTPA programs, resulting in notable 
differences between urban and rural SDAs. 

Within the local context, we also found 
substantial variation in program design that 
reflected SDAs' choices of goals for the 
JTPA program. Generally, we found SDAs 
to be influenced by three different objectives: 
(1) commitment to serving specific types of 
clients; (2) commitment to responding to 
local employer needs and interest; and (3) 
commitment to achieving specific levels on 
JTPA perforaiance standards. The relative 
emphasis that SDAs placed on these three ob- 
jectives varied a great deal. 

To the extent that SDAs stressed client 
objectives, they tended to design program ser- 
vices appropriate to the needs of those clients 
and to choose management practices to en- 
sure achievement of their client goals. To the 
extent that SDAs stressed employer needs, 
they gave priority to designing and im- 
plementing a service mix to train individuals 
for the more highly skilled jobs and enrolled 
participants appropriate for that training. 

In general, most client-oriented and 
employer-oriented SDAs were able to meet 
their performance goals without major 
revisions to then- other goals. In conjunction 
with clearly identified client and service 
goals, performance standards appeared to 
have their intended effects of increasing ef- 
ficiency and accountability. 
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This evaluation found only a few SDAs 
that gave priority to their perfoimance goals 
(generally goals of maximizing incentive 
awards) at the expense of other program ob- 
jectives. Only in those SDAs without well- 
defined client or service goals did 
performance standards have notable unin- 
tencled effects. In a vacuum, performance 
stanclaids can produce unintended effects 
of recucing service to the hard to serve and 
decreasing the intensity of services. 

Effects of State Policies 

The JTPA legislation gives considerable 
authority to the States, both to establish per- 
formance-standards policies and to set 
priorities for the types of clients to be served 
in JTPA programs. There is a great deal of 
variation in how States have chosen to use 
that authority. This evaluation examined how 
SDAs responded to these differences in State 
policies. 

We found that Slates can provide an ef- 
fective leadership role in setting client 
priorities for JTPA programs. States that es- 
tablished target groups for JTPA programs 
did influence SDAs to enroll more hard-to- 
ser/c clients. Often States used performance- 
standards policies to convey their client 
priorities, including setting State standards 
for service to target groups or using 6% 
funds for service to those groups. States that 
used the optional DOL adjustment models, 
which adjust standards for the types of clients 
served, also influenced SDAs to enroll more 
hard-to-serve clients. 

Some of the ways that States have used 
their authority in establishing performance- 
standards policies have had unintended ef- 
fects. In particular, incentive policies that 
emphasize exceeding standards rather than 
simply meeting standards tend to reduce ser- 
vice to some haid-to-serve groups and reduce 
the amount of basic skills remediation ser- 
vices. Emphasis on exceeding standards 
seems to have institutionalized sometimes in- 
tense competition among SDAs and con- 



veyed the message that "cheaper is better.** 
The effects of these policies, however, are 
not large and affect SDA choices at tiie 
margin. 

Effects of Federal Policies 

Federal performance-standards policies 
are intended to be neutrd with respect to the 
types of clients served. They are intended to 
guide SDAs to choose cost-effective services 
but not to reduce local flexibility in designing 
services appropriate to local needs. This 
evaluation found that the Federal standards 
for the entered-employment rate and wage 
rate for adults generally did not have unin- 
tended effects on clients or services. Further- 
more, no evidence showed that the welfare 
entered-employment-rate standard inhibited 
service to welfare recipients. 

The Federal cost standards, however, had 
the most unintended effects and were the 
least comparably measured of all the Federal 
performance measures. This evaluation found 
that SDAs in States that placed more weight 
on the Federal cost standard tended to serve 
fewer hard-to-serve clients and that SDAs 
concerned about exceeding the cost standards 
tended to design less-intensive services. At 
the same time, this evaluation found serious 
measurement problems with the cost stand- 
ards. We found large differences in the extent 
to which SDAs were leveraging JTPA funds, 
either by using funds from other programs to 
help fund JTPA Title II-A programs or by 
using service providers that had alternative 
funding sources. As a result, it is difficult to 
compare the cost of services received by 
JTPA participants across SDAs. 

There was also considerable confusion 
among SDAs in how to manage youth 
programs to meet all the youth standards. Be- 
cause the goals of in-school and out-of- 
school programs were so different, SDAs that 
emphasized one type of service often found it 
difficult to meet at least one youth standard. 
Thus, the Federal youth standards tended to 
constrain SDA choices about serving in- 
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school versus out-of-school youth. 
Summary 

This evaluation found that performance- 
standards policies can' influence SDAs' 
choices about clients, services, and costs. 
Some effects are the intended results of State 
leadership in setting priorities for the JTPA 
program; some effects are the unintended 
results of State and Federal policies that tend 
to reduce service to hard-to-serve groups or 
reduce the intensity of service. The unin- 
tended effects, however, are neither large nor 
inevitable. Performance standards do not 
preclude SDAs from enrolling hard-to-serve 
clients or from providing intensive services 
but do affect some SDA tradeoffs at the mar- 
gin. Furthermore, tiiere is substantial varia- 
tion in how SDAs choose to react to 
standards. Most SDAs can meet their perfor- 
mance goals witiiout m.ajor revision to their 
client and service goals. Only in SDAs 
without client or employer goals do perfor- 
mance standards have substantial unintended 
effects. 

Detailed Findings 

Overview of Evaluation Design 
To conduct a comprehensive and 
thorough evaluation of performance stand- 
ards, this study was composed of two com- 
ponents, a quantitative evaluation based on 
surveys of all States and SDAs and a qualita- 
tive evaluation based on indepth case studies 
of 8 States, 30 SDAs, and 87 service 
providers. 

The quantitative evaluation examined 
whether differences in State performance- 
standards policies affected SDAs* choices 
about the types of clients to be served, the 
services provided, or the program costs. This 
analysis provided objective, statistically valid 
evidence of the impacts of State performance- 
standards policies for the system as a whole. 
The analysis also examined the influence of 
SDA enrollment criteria and service-provider 
arrangements on clients, service, and costs. 



Data for the quantitative", component was 
obtained from the following sources: 

• Review of all State performance- 
standards policies for PY 86 to deter- 
mine incentive and sanction policies, 
adjustment procedures, additional 
State performance standards, and 
policies for serving the hard to serve. 

• Survey of all SDA directors to obtain 
information on PIC and local elected 
official (LEO) concerns and in- 
fluence, intake procedures, enroll- 
ment criteria, and service-provider 
arrangements. (The response rate to 
this survey was 87%.) 

• Survey of fiscal/management informa- 
tion system (MIS) staff to obtain in- 
formation on resources, expenditures, 
and number of participants receiving 
services in various program activities 
in PY 86. (The response rate to this 
survey was 74%.) 

• Census data on the characteristics of 
the eligible population in each SDA 
as well as other published data on 
local characteristics, including the un- 
employment rate, percentage of 
population with income below pover- 
ty, population density, and area wage 
levels. 

• JTPA Annual Status Report (JASR) 
data on the characteristics of in- 
dividuals served in JTPA programs 
and program expenditures for PY 86. 

The analysis estimated ihe impacts of 
State policies and SDA practices on clients, 
services, and costs, controlling for charac- 
teristics of the local economy and of the local 
eligible population. 

The second component of this evaluation 
was a qualitative analysis, based on indepth 
case studies that included extensive onsite in- 
terviews and observations. The purpose of 
this component was to explore the influence 



of Federal performance-standards system on 
States and SDAs and to supplement the quan- 
titative analysis in understanding the extent 
to which State performance-standards 
policies influenced SDAs. The qualitative 
analysis also examined the mechanisms by 
which performance-standards policies af- 
fected clients, services, and costs and iden- 
tified how performance standards interacted 
wdth other influences in determining State, 
SDA, and service-provider behavior. 

The qualitative component was based on 
a sample of 30 SDAs and 87 service 
providers in 8 States. Although necessarily 
limited in number, these agencies were 
chosen to be as representative as possible of 
the JTPA system as a whole. 

• The eight States included California, 
Maryland, Massachusetts, Missouri, 
New Mexico, New York, North 
Carolina, and Ohio. These States 
were chosen on a random basis that 
also ensured variation in State perfor- 
mance-standards policies and 
geographic region. 

• The 30 SDAs were selected, 4 within 
each State (except in New Mexico, 
where 2 of the 3 SDAs were 
selected), on a random basis that also 
ensured variation in SDA perfor- 
mance relative to their model-ad- 
justed standards. 

• Eighty-seven service providers were 
chosen, approximately three per 
SDA. These providers were selected 
purposively to ensure variation in the 
type of provider, type of service, and 
type of contract. 

Extensive interviews were conducted 
with State Job Training Coordinating Coun- 
cils (SJTCC) and agency staff at the State 
level, with PIC representatives, SDA direc- 
tors, and staff at the SDA level, and with 
program administrators and direct service 
staff at the provider level 



Results from the Quantitative Analysis 

The quantitative analysis identified how 
variations in State performance-standards 
policies have affected the clients, services, 
and costs of local JTPA programs. The quan- 
titative analysis traced the effects of State per- 
formance-standards policies in two ways. 
First, we measured the total effects of State 
policies on clients, services, and costs, con- 
trolling for a variety of other variables, in- 
cluding local economic conditions, 
characteristics of the JTPA-eligible popula- 
tion in each local area, and variations in the 
roles played by PIC members and LEOs. 
Second, we examined the impact of several 
SDA design decisions on the types of clients 
served including procedures to recruit and 
enroll different type of clients, criteria for 
selecting clients, the type of service providers 
used, and the type and terms of contracts. We 
then examined the influence of performance- 
standards policies on those design decisions. 

The analysis of the impact of perfor- 
mance-standards policies showed that several 
dimensions of State performance-standards 
policies do significantiy affect the extent to 
which SDAs serve members of hara-to-serve 
groups and the types and average duration of 
JTPA services provided in the local area. 
Policies that increase service to hard-to-serve 
clients in JTPA include the following: 

• Use of the POL adjustment model 
significantiy increases the percentage 
of both adult welfare recipients and 
dropouts served. Use of tiie adjust- 
ment model also significantiy in- 
creases enrollment of youth with 
"other barriers to employment" (that 
is, offenders, handicapped in- 
dividuals, and limited-English 
speakers). 

• State policies on adjustments beyond 
the model have less consistent effects 
but increase services to adult 
minorities and increase services to 
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youth welfare recipients and 
minorities. 

• State policies on use of 6% for hard 
to serve (setting aside 6% funds for 
services to hard-to-serve groups, re- 
quiring that some 6% funds be used 
to serve hard-to-serve groups, or ex- 
empting 6%-funded projects from the 
performance standards if they are 
used for hard-to-serve groups) in- 
crease service to adult welfare 
recipients and to adults with "other 
employment barriers" but decrease 
services to women. For youth, these 
policies increase enrollment of wel- 
fare recipients and in-school youth. 

• State policies encouraging services to 
specific client groups (a formal policy 
statement or a 6% policy that rewards 
service to a particular group) are ef- 
fective in increasing enrollment of 
welfare recipients, dropouts, and 
older adults. 

Policies that decrease service to hard-to- 
serve clients in JTPA include the following: 

• Incentive policies tliat emphasize ex- 
ceeding standards include (1) reserv- 
ing most of tlie incentive awards for 
substantially exceeding standards, (2) 
having SDAs compete for 6% funds 
by comparing performance levels 
across SDAs, (3) not "capping" the 
performance level beyond which addi- 
tional incentives would not be earned, 
and (4) requiring a large number of 
standards be exceeded to qualify for 
incentive awards. These policies sig- 
nificantly reduce service to welfare 
recipients and minorities for both 
adults and youth and reduce service 
to older adults and in-school youth. 

• Placing greater weight on the cost 
standards reduces service to adult wel- 
fare recipients and adult dropouts. 
For youth, this practice was estimated 



to reduce services to welfare 
recipients, dropouts, and youth with 
other bairiers to employment 

These results indicate tiiat some State 
policies have their intended effects bu't that 
the incentive policies in some States are unin- 
tentionally reducing service to some hard-to- 
serve groups. 

State performance-standards policies also 
influence the types of services offered in 
JTPA programs, including tiie relative em- 
phasis on types of program activities and the 
average duration of JTPA services. 

Several State policies are generally as- 
sociated with more intensive or longer term 
services: 

• Adjustment procedures . Use of the 
DOL model significantiy increases 
the length of adult programs and 
reduces the amount of job-search as- 
sistance provided to youth. State 
policies that specify procedures for 
additional adjustments significantiy 
increase provision of basic skills train- 
ing to youth and reduce job-search as- 
sistancfc, although they have an 
unexpected negative impact on the 
length of participation. 

• State policies for hard-to-serve 
groups . State policies for serving wel- 
fare recipients significantiy increase 
the amount of basic skills training of- 
fered for both adults and youth and 
reduce the amount of on-the-job train- 
ing (OJT), significantiy so for adults. 
Policies for serving dropouts sig- 
nificantiy increase the average length 
of services for adults and youth, but 
policies for serving dropouts also in- 
crease the provision of job-search as- 
sistance, reducing OJT for adults and 
work experience for youth. 

• Placing greater weight on the wage 
standard . Policies emphasizing the 
wage standard significantiy increase 
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provision of classroom training in oc- 
cupational skills and reduce OJT for 
adults. 

State policies that were found to reduce 
service to haid-to-serve groups also affected 
program services. These policies generally 
either reduced the intensity of services or in- 
creased the employment focus of the pro- 
gram. These policies include the following: 

• Emphasis on exceeding standards . 
State policies that emphasize exceed- 
ing standards lead SDAs to provide 
less basic skills training and more 
classroom training in occupational 
skills for both adults and youth. For 
youth, these policies also reduce pre- 
employment/work maturity training 
and reduce the average length of 
program participation. 

• Placing greater weight on the cost 
standards . State incentive policies that 
place greater weight on cost standards 
reduce the average length of services 
for adults and increase the provision 
of pre-employment/work maturity 
training for youth. 

Finally, State performance-standards 
policies appear to have fewer effects on 
program costs than on clients or services: 

• Adjustment policies . Use of the DOL 
models has no signiticant effects on 
program costs, although State proce- 
dures to allow for adjustments sig- 
nificantly increase the amount spent 
per terminee for both adults and 
youth. 

• Emphasis on exceeding standards . 
State policies that emphasize exceed- 
ing performance standards increase 
costs per terminee, significantly for 
adults, although not cost per entered 
employment or cost per positive ter- 
mination. Thus, these policies lead 
SDAs to enroll less hard-to-serve 
clients but to provide them with more 



classroom training in occupational 
skills, the most expensive service. 

• Placing greater weights on cost stand- 
ards . The weights placed on the cost 
standards do not significantly affect 
costs per terminee or SDA perfor- 
mance on the cost standards. 

One reason that the quantitative results 
are smaller for costs may be that there is 
serious noncomparability of reported costs 
across SDAs. Some SDAs extensively 
leverage JTPA resources with funds from 
other programs, as discussed in the qualita- 
tive analysis. 

The second phase of the quantitati\ ^ 
analysis examined the mechanisms through 
which the State performance-standards 
policies influenced SDA behavior. In tht> 
first step of this examination, we examined 
the influence of a variety of SDA implemen- 
tation practices on clients, services, and 
costs. In the second step, we examined the 
relationship between the State performance- 
standards policies and those SDA practices 
that appear to influence clients, services, and 
costs. 

SDA policies and practices that were 
found to affect the types of clients served by 
JTPA include the following: 

• PIC influence . PIC influence per se 
does not reduce service to the hard to 
serve. The results suggest that PICs 
that see their role as guiding the 
design of the program are associated 
with greater enrollment of the hard to 
serve while PICs that are more in- 
volved in contracting are associated 
with less service to some hard-to- 
serve groups. 

• Enrollment criteria . SDAs that use ob- 
jective basic skills or educational 
criteria serve more hard-to-serve 
clients, probably because these 
criteria are used to slot participants 
into appropriate activities. In contrast, 
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SDAs that use subjective judgments, 
require previous work histories, or ac- 
cept "reverse referrals" from 
employers for OJT slots serve sig- 
nificantly fewer hard-to-serve clients. 

• Service-provider arrangements . The 
use of community-based organiza- 
tions (CBOs) as service providers is 
associated with greater service to 
several hard-to-serve groups, includ- 
ing both adults and youth dropouts. 
The percentage of expenditures in per- 
formance-based contracts does not 
reduce service to hard-to-serve 
groups, although there is a weak pat- 
tern that setting stringent contract 
terms reduces the service to some 
hard-to-serve groups in JTPA 
programs. SDAs that vary the terms 
of their contracts serve significantly 
more adult welfare recipients and 
dropouts. 

• Program services . The t>'pes of 
program services offered by SDiVs 
have a strong influence on the types 
of clients enrolled in JTPA programs. 
Basic skills remediation has the 
strongest association with enrollment 
of hard-to-serve groups, followed by 
classroom training in occupational 
skills, then job-search assistance. * 
SDAs that provide more OJT tend to 
serve significantly fewer hard-to- 
serve clients. 

After identifying the different effects that 
these various SDA designs and implementa- 
tion practices have had on client outcomes, 
the quantitative analysis examined whether 
the practices tiiemselves were related to varia- 
tions in State performance-standards policies. 
The findings are consistent with the overall 
relationships between State policies and 
client patterns and help elucidate the 
mechanisms by which these relationships 
are realized. 



SDAs in States that use the DOL model, 
which was positively associated with ser- 
vices to hard-to-serve clients, were less likely 
to use subjective judgments as enrollment 
criteria or to allow employers to preselect 
OJT trainees, two practices found to dis- 
courage services to hard-to-serve clients. Fur- 
thermore, SDAs in these States were more 
likely to target both youth and adult services 
to welfare recipients and groups with other 
barriers to employment 

SDAs in States with specific policies en- 
couraging services to welfare recipients were 
more likely to establish procedures to recruit 
welfare recipients for both adult and youth 
participants. On the other hand, the local 
decision of whether to target dropouts was 
not significantly affected by State policies en- 
couraging service to d'^DOuts. 

SDAs in States wiu. policies that em- 
phasize exceeding the standards, which were 
associated with reduced levels of services to 
hard-to-serve groups, were more likely to use 
subjective judgements about the likelihood 
of completing as a client enrollment 
criterion, which was also associate^ with 
reduced levels of services to hard-to-serve 
clients. These SDAs also made increased use 
of for-profit providers, which reduced the 
levels of service to some hard-to-serve 
groups. Furthermore, SDAs in States with a 
r^ffong emphasis on exceeding the standards 
used more performance-based contracts with 
higher "holdbacks" for placements and 
higher wage rates. It is not clear from the 
quantitative analysis that these last two prac- 
tices had a direct impact on clients ocv/td by 
the JTPA program, although higher required 
wage rates did appear to discourage services 
to some hard-to-serve groups. 

SDAs in States that give greater than 
average weight to the cost standards, which 
was associated with reduced services to hard- 
to-serve groups, also were more likely to 
allow employers to make prescrcened refer- 
rals of clients for OJT slots and were less 
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likely to use special procedures to recruit wel- 
fare recipients into the JTPA program. 

Overall, these results indicate that perfor- 
mance-standards policies affect the types of 
clients served, in part through their influence 
on the types of services offered and in part 
through their influence on SD A enrollment 
and contracting practices. 

The results of the quantitative analysis in- 
dicate several State performance-standards af- 
fect SDA decisions about whom to serve and 
the types of services to provide. These results 
are statistically reliable and have important 
policy implications. Nonetheless, tb^ effects 
of these policies are not large. These policies 
do not preclude service to the hard to serve or 
prevent provision of intensive services. 
Rather, these policies affect SDA tradeoffs at 
the margin. 

Results from the Qualitative Analysis 
The analysis of the qualitative case study 
data examined how and why the 30 case 
study SDAs varied in their reactions to the 
performance standards. In tracing the effects 
of the Federal and State perforaiance-stand- 
ards policies facing each SDA, we examined 
several elements of SDA program-design and 
implementation decisions: (1) the develop- 
ment of client goals and SDA plans for ser- 
vices to particular groups; (2) the develop- 
ment of overall service designs, including 
what services to offer, how much of each ser- 
vice to provide, how intensive to make each 
service, and how services relate to each other 
in the development of service plans for in- 
dividual clients; and (3) the design of 
management practices for selecting service 
providers and overseeing program implemen- 
tation. Within each of these program ele- 
ments, we used the case study data to address 
questions about how the performance stand- 
ards influenced SDA and service-provider 
decisions, how the performance standards in- 
teracted with other factors, and how the 
resulting SDA practices influenced clients, 
services, and costs. 



Effects on Local Program Objectives. 

There was considerable variation both in the 
extent to which the performance standards in- 
fluenced local performance goals and in the 
ways that standards produced these effects. 
About one-thiixl of the case study SDAs had 
a goal of meeting the standards and "staying 
out of trouble" (that is, avoiding sanctions). 
Another one-fourth of the SDAs indicated a 
goal of exceeding the standards slightly, to 
have a performance safety margin and to real- 
ize some incentive awards. The remaining 
sites indicated a goal of performing at as high 
a level as they could on the standards, 
primarily because it was important to maxi- 
mize their potential incentive awards. Pride 
was also mentioned by all types of SDAs as 
an important incentive to perform at or above 
the level of performance standards. 

The relative importance of the perfor- 
mance standards in shaping local goals and 
priorities was conditioned in part by the im- 
portance to the SDA of receiving the maxi- 
mum amount of incentive funds. Some SDAs 
placed high priority on receiving funds be- 
cause they viewed them as compenl>.•^ting for 
shrinking 78% allocations and tight ad- 
ministrative limits. Others valued the awards 
for their public relations value. Still others 
used the awards to undertake projects that 
they could not fund with 78% money, such 
as projects serving high-risk clients or 
marketing or outreach projects that only in- 
directly contributed to outcomes. 

Several typologies of responses to the per- 
formance standards help explain the varia- 
tions in how the perfomiance standards 
influenced clients, services, and costs within 
the case study sample. Client-oriented SDAs 
gave priority to their client objectives and 
oriented their program-design and manage- 
ment decisions around how this would affect 
their client goals. Employer-oriented SDAs 
gave priority to designing and implementing 
a service mix that would train individuals for 
jobs in demand in the local labor market. Per- 



11 21 



fonna nce- oriented SD As gave priority to 
meeting or exceeding the performance stand- 
ards (or other State or local performance 
measures). Hiese wei^ SDAs without strong 
employer or client goals to counterbalance 
their goals to exceed their performance 
standards. 

Some client-oriented or employer- 
oriented SDAs were indifferent to the stand- 
ards when they designed their programs, 
either because they did not care if they me * 
the standards or because the standards were 
so easy to meet or exceed. Others adopted 
risk management strategies that enabled them 
to meet their performance goals while still 
furthering local client and service objectives. 
Performance-oriented SDAs tended to adopt 
risk avoidance strategy "^s 'hat increased the 
likelihood that they w.^uld meet the perfor- 
mance standards by avoiding hard-to-serve 
clients or adjusting the ser\'ice mix for 
reasons that were standards oriented. 

Finally, SDAs varied in the extent to 
which the SDA assumed respon jibility for 
meeting its performance goals, shared that 
responsibility with service providers, or 
passed on the primary responsibility for per- 
formance to the service-provider level. 

In balancing these different goals, the per- 
formance standards played an important role 
in some SDAs and a minor role in others. 
Only a few SDAs in the case study sample 
purely relied on "risk avoidance" rather than 
"risk management" in their response to the 
performance standards. Most SDAs could 
meet their performance goals without having 
to resort to a major revision of other 
program goals. 

Effects on Client Goals^ The qualitative 
analysis suggested that the performance 
standards do not influence SDA client goals 
directly but rather influence clients served 
through their influence on SDA service 
designs and implementation practices. Other 
factors Uiat did influence client goals in- 
cluded (1) State equitable-service require- 



ments and State policies for serving specific 
hard-to-servw groups; (2) die 40% youUi ex- 
penditure requirement; and (3) Uie charac- 
teristics of die applicant pool, which were 
strongly influenced by environmental factors 
beyond the SDA's control, particularly die 
local unemployment rate. 

Moreover, the case studies demonstrated 
Uiat the performance standards did not 
prevent SDAs that had a strong commitment 
to serving hard-to-serve groups from target- 
ing and: zrving those groups. Welfare 
recipients and dropouts were served out of 
proportion to their incidence in the popula- 
tion in a number of SDAs. Strategies for serv- 
ing these clients included (1) designing 
special training classes that integrated 
remediation and world-of-work orientation 
wiUi occupation skills curricula, (2) leverag- 
ing odier XrPA funds (for example, 8% and 
6%) or non-JTPA funds (for exair-ple. State 
funds and foundation funds), (3) targeting 
employers that offered healdi benefits, and 
(4) making explicit tradeoffs to balance ser- 
vices to hard-to-serve clients with less expen- 
sive services to other groups to enable die 
SDA to meet its performance requirements. 
In addition, gn^ups widi other barriers to 
employment (the handicapped, offenders, 
and limited-English speakers) were targeted 
for services by some SDAs. These groups 
were usually served in special programs 
operated by providers widi a history of work- 
ing with these groups. 

Despite this general pattern, some SDAs 
that were highly motivated to exceed their 
standards were influenced by th ^"?xfor- 
mance standards to develop ") isk avoid- 
ance" strategies. These SDAs tended to ad- 
dress die needs of some subset of eligible ap- 
plicants diat did not include those with the 
greatest barriers to employment. 

Effects on Program Services and 
Costs* Performance standards had slightly 
more influence on die service mix offered to 
adults by SDAs. A few SDAs mentioned the 
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standards when explaining their emphasis on 
OJT rather than classroom training. 
However, at least equally important influen- 
ces on service mix were (1) the SDA's his- 
torical pattern of service delivery, as evolved 
ftom the Comprehensive Employment and 
TYaining Act (CETA) program; (2) the 
SDA's intentions about serving different 
client groups; and (3) PIC preferences about 
what services to emphasize in the SDA's ser- 
vice package. 

SDAs exhibited wide vanation in the in- 
tensity and duration ofJTPA services. In ad- 
dition, the costs'ofTrPA^srervi^es'v^ari'ed— — 
dramatically among SDAZy and average cost 
figures often disguised tremendous variation 
within an SDA in the duration and intensity 
of services received by different participants. 
The cost standards had some influence on 
this aspect of service design, because SDAs 
often wanted not only to meet but to exceed 
these standards to maximize their incentive 
awards. For the SDAs that were highly 
motivated to "overperform" on tlie standards, 
the standards influenced them to design ser- 
vice programs that had short durations, 
yielded high entered-employment rates, and 
resulted in low unit costs. Hius, in these 
SDA?, the initial reaction to the standards 
was to emphasize quantity over quality. A 
number of these SDAs are beginning to shift 
toward more intensive services, however, par- 
tially in response to the implementation of 
the follow-up standards. 

However, other factors were equally as 
important in determining the intensity and 
duration of services. These included (1) the 
availability of other funds to supplement 
Title II-A funds in purchasing services to 
benefit Title II-A enrollees, (2) the cost of the 
various service packages available from the 
service providers in the SDA, (3) the SDA's 
goals about the types of clients it wanted to 
serve and the intensity and types of services 
those clients required, and (4) the desire to 
spread JTPA funds over as large a client pool 



as possible. 

There were two reasons why SDAs were 
very concerned about reducing average 
JTPA costs. A majority of SDAs visited were 
experiencing declining JTPA allocations (due 
to declining unemployment rates), which re- 
quired a decision about whether to serve a 
smaller volume of participants and keep the 
average cost of services stable or to try to 
serve as many participants as possible by cut- 
ting back the average cost of the services 
received. Almost universally, the SDAs 
visited had decided to try to reach as many in- 
idividuals'asi)ossible, without diluting the ser- 
vice intensity beyond reason. 

The second reason for an emphasis on 
cost reduction was the SDAs' desire to ex- 
ceed the cost performance standards. Al- 
though the vast majority were in no danger of 
missing these standards, the emphasis on 
overperforming grew out of a desire to maxi- 
mize their incentive awards. 

The existence of performance standards 
for youth did not seem to affect the extent of 
SDA resources devoted to youth programs 
perse. Performance standards did, however, 
affect SDA decisions about serving in-school 
versus oiit-of-school youth. There is an inten- 
sive debate among and within SDAs about 
the extent to which JTPA programs should 
be offering employment-oriented or com- 
petency-oriented training. SDAs that chose to 
emphasize competency-oriented services 
found it difficult to meet their entered- 
employment-rate standards while those em- 
phasizing employment-oriented programs 
found it difficult to meet their positive- 
termination-rate standards. 

Nonetheless, local goals about whether to 
run employment-oriented or competency- 
oriented programs for youth, as \vell as the 
Federal requirement to spend 40% of JTPA 
funds on youth, appeared to have greater in- 
fluences than the standa^uS. However, a num- 
ber of SDAs had instituted a youth employ- 
ment-competency system to meet their posi- 
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tive-tennination-rate standards. 

Effects on Management Practices. 
SDA management practices were designed to 
further a variety of management goals, in- 
cluding ensuring (1) that the planned mix 
of clients was actually served, (2) that the 
planned mix of ':2rvices was actually 
provided, (3) that each client was referred to 
appropriate services, (4) that service 
providers enrolled clients that were 
appropriate for that service and had a 
reasonable likelihood of success, (5) that 
costs would be kept within reasonable limits, 
and (6) that the desired performance levels 
would be achieved. 

To further their management goals, 
SDAs instituted a number of practices for en- 
rollment and assignment to services, place- 
ment, and selection of service providers. 
Some of these practices grew out of their ap- 
proach to the performance standards. 

Most of the SDAs visited maintained 
centralized control over the outreach, assess- 
ment, and referral of clients to service com- 
ponents, with a smaller number leaving 
responsibility for these services to service 
providers. Contrary to expectations, the en- 
rollmentofhard-tO'Serye clients was not cor- 
related with the SDA's control over enroll- 
ment; SDA and service-provider goals about 
serving the hard to serve seemed to be more 
important than who controlled enrollment. 

Some SDAs protected themselves against 
the risk of serving hard-to-serve individuals, 
however, by delaying enrollment until after 
some services had been delivered to avoid 
having to include early dropouts in their 
statistics. 

Performance standards did affect SDAs' 
service-provider arrangements. In particular, 
service providers* past performance often in- 
fluenced the size of their contract in the next 
funding period. However, a number of addi- 
tional factors influenced provider selection 
and retention, often more strongly than the 
performance standards per se, including the 



availability of alternative service providers, 
the characteristics of the local economy, the 
amount and categories of services that the 
SDA contracted out, and the goals and 
philosophy of the PIC or SDA. 

Two-thirds of the 'Hse-study SDAs had 
some current perforL'i mce-based contracts. 
Although the performance standards were a 
factor in the spread of performance -based 
contracting among the case study SDAs, the 
15% administrative limit was at least as im- 
portant; this was especially true in the con- 
text of declining Title II-A allocations. 

Among performance-based and cost-reim- 
bursement contracts alike, the great majority 
incorporated performance criteria as either re- 
quirements or goals. In about half of the case 
study SDAs, contracts built in the SDAs' 
standards (usually with a "cushion") direcdy. 
Most of the others varied key contract terms 
among their providers but took care that, ag- 
gregating across the providers, they met or 
exceeded the SDA's own standards. 
Providers facing high-placement expecta- 
tions, high-wage goals, high-placement 
holdbacks, and caps on maximum service 
duration were inclined to be cautious about 
the clients they would accept. If high-percent- 
age-holdback contracts become more 
prevalent, "riskier" die its, who have more 
severe employment bar iers, may be more 
likely to be screened oat of service programs. 

Contract terms should not, however, be 
understood as simple outgrowtlts or reflec- 
tions of the standards. Many of the key fea- 
tures of these contracts predate JTPA 
programs, as does the use of performance- 
based contracts in several of the case study 
SDAs. The relationship between contract 
terms and the SDA's performance standards 
was affected by several factors. These in- 
cluded how difficult a given SDA found 
recruiting and retaining clients, the relative 
employability of available clients, the 
availability of alternative service providers, 
the status and direction of the SDA*s Title II- 
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A allocation, its access to non-JTPA funding, 
and the philosophy and goals of the PIC and 
the SDA staff. Finally, even when the form 
of performance-based contracts is a recent 
development, they often essentially continue 
the mutual expectations that an SDA and a 
provider had developed in earlier years. 

Service providers varied in their reactions 
to the performance expectations placed on 
them by the SDAs. In a few cases, contrac- 
tors faced pressures: to perform at high levels, 
which caused greatei caution in client selec- 
tion. Most providers, however, had a commit- 
ment to serving hard-to-^erve clients, and 
their contract terms gave .hem enough 
flexibility to meet that commitment. 

For the most part, the management prac- 
tices found in the case study SDAs are consis- 
tent with "risk management" strategies 
rather than ''risk avoidance" strategies. That 
is, using these practices to safeguard perfor- 
mance levels, SDAs could and did serve dif- 
ficult clients under the JTPA program while 
attaining their performance goals. 

Implications for the Performance- 
Standards System 

The findings of this evaluation indicate 
that the performance standards do not 
preclude SDAs from enrolling hard-to-serve 
clients or from providing intensive services. 
Nonetheless, the results of this evaluation 
also indicate that not all aspects of the perfo- 
mance-standards system are working as in- 
tended and that some policies have unportant 
unintended effects. In this section, we ex- 
amine the implications of the results for both 
Federal and State performance-standards 
policies. 

Federal Policies 

Choice of Performance Measures. 
Cost Measures . The JTPA legislation man- 
dates that cost standards be included as 
Federal perfonnance standards. Both the 
quantitative and qualitative results, however, 
indicate that the cost standards had the most 



unintended effects and were the least com- 
parably measured of all the Federal stand- 
ards. There were unintended effects on both 
clients and services: States that placed a high 
weight on the cost standard lead SDAs to 
serve fewer welfare recipients and dropouts; 
in our case study sample, some SDAs con- 
cerned about exceeding their cost standards 
designed short-term, less intensive services. 

The measurement problems in the cost 
standards were also serious. SDAs that ware 
able to leverage JTPA Title II-A funds with 
other program funds or that were able to rely 
heavily on service providers with alternative 
funding sources had much lower measured 
costs than those SDAs that relied solely on 
Title II-A resources to train their participants. 
Some quantitative results supported this find- 
ing, and our case studies found dramatic dif- 
ferences in the leveraging of JTPA funds. As 
a result, basing incentive payments on the 
cost standards probably rewards differences 
in the local availability of other training 
resources as much as differences in manage- 
ment quality. 

Clearly, cost-effectiveness is an essential 
goal of any program, and an increase in the 
extent of leveraging, which reduces redun- 
dant services, may be a :Iesired effect of the 
cost standard. Nonetheless, we found that 
many SDAs were already very concerned 
about costs because of a strong desire to 
serve as many participants as possible vith 
limited (and often declining) JTPA funds. 

The results of this evaluation suggest that 
alternatives to the cost standards should be 
explored. Out of concern for tiie unintended 
effects of tiie cost standards, DOL set much 
more lenient costs standards for PY 88. This 
policy is not likely to be effective, however, 
in States that strongly en^phasize exceeding, 
not just meeting, standards, particularly when 
the cost standards are weighted more heavily. 
An alternative ti^at requires legislative chan- 
ges would be to set maximum costs per 
entered employment (or positive termination) 
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that SDAs could spend but not to base incen- 
tive payments on cost standards. In effect, 
this policy would treat costs as a compliance 
rather than a ]:erformance-standards issi 
Youth Measures . The only other problem in 
the choice of measures was a less serious con- 
cern raised about the youth measures. SDAs 
were generally confused about how to 
manage the youth programs to meet both the 
entered-employment-rate and positive-ter- 
mination-raie standards. Because the goals of 
in-sc.*uol and out-of-school programs were 
so different, SDAs that emphasized one type 
of ser\'ice often felt pinched by at least one 
youth standard, 0ns possible solution is :o 
set separate standards for in-school programs 
aiid out-of-school programs so that the 
SDAs* choice of program mix would not af- 
fect their ability to meet the youth standards. 

A related issue is the inclusion of youth 
employment competencies in the positive-ter- 
mination measure. Although evaluating the 
quality of the competency systems was 
beyond the scope of this Gtudy, we found that 
a substantial majority of the case study SDAr 
had made a conscientious effort to establish 
meaningful systems. However, many SDAs 
reported that they would not have adopted 
the competency-based approach in the ab- 
sence of th'a? positive-termination-rate stand- 
ard. Thus, SDAs naturally tended to develop 
systems that would reflect well on the 
programs that they were running. Despite the 
temptation to implement a superficial youth 
employment competencies (YEC) system, 
only 2 of the 30 case study SDAs had 
adopted a "quick and dirty" competency sys- 
tem, and 1 of those used it only as a back-up 
outcome for their employment-oriented 
youth services. 

Level of Standards. The results of this 
evaluation do not indicate the neeJ for chan- 
ges in the general levels at which perfor- 
mance standards are set. The low levels of 
most standards generally allow SDAs to 
meet the standards and design theu- programs 



for local needs, while providing a clear incen- 
' t for good performance The only excep- 
tion is a concern among a number of SDAs 
that the positive-termination-rate standard is 
too hi^^,h for serving very at-risk youfh, many 
of whom may drop out of the program. 

DOL Adjustment Models* The use of 
the DDL adjustment models appears to be 
very effective in increasing service to hard-to- 
serve groups and in increasing the length of 
adult servTcesT^'These models, therefore, ap- 
pear to have their intended effects of holding 
SDAs harmless for then: local decisions of 
whom to serve in JTPA programs. 

The models adjust for :;haracteristics of 
clients served aad the length of services, as 
well as local economic conditions. The 
models do not, however, adjust for the typ:;S 
of services offered. This evaluation has 
found that the types of services offered sig- 
nificantly afffcr,t the types of clients enrolled 
in JTPA programs and tliat some of th^ unin- 
tended effects of performance standards, par- 
ticularly the con standards, are to reduce the 
provision of intensive services. Furthermore, 
we found that the types of services sig- 
nificantiy affect program costs, with class- 
room training in occupational skills and basic 
skills remediation costing considerably more 
than job-search assistance. 

Because the adjustment models are effec- 
tive tools in holding SDAs harmless for local 
decisions, these results imply that serious 
consideration should be given to including 
some adjustments for program activities in 
the models for the cost standards. In par- 
ticular, adjucting for basic skills remediation 
would enable SDAs to spend more money 
when they include basic skills remediation in 
their service package. 

Technical Assistance* The evaluation 
findings suggest that additional technical as- 
sistance from DOL could be used to improve 
the effectiveness of the performance-stand- 
ards system. First, DOL could provide assis- 
tance to States to develop their full 
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leadership potential in setting client and ser- 
vice priorities for the'^ SDAs and to develop 
performance-standaras policies consistent 
with those goals. Second, DOL could provide 
additional assistance to SDAs to develop 
strategies to balance performance goals with 
other client and service goals. Third, al- 
though there is a moderate level of under- 
standing of the adjustment model, many 
SDA respondents felt unsure that they fully 
understood how the process worked. Given 
how important tlie model is in encouraging 
service to the hard to serve, DOL efforts to 
improve understanding of the model should 
increase the intended effects of the model. 

Reporting. We found very littie evidence 
that the Federal reporting requirements neces- 
sary to support the adjustment models were a 
burden to SDAs, providers, or participants. 
Most of the information required was infor- 
mation that the providers needed to manage 
their programs or for equal-opportunity 
documentation. Furthermore, most SDAs 
maintained computerized participant-level 
data bases (often supported by the State), so 
preparing reports on participant characteris- 
tics was not difficult. In fact, some respon- 
dents indicated tiiat die greatest burden was 
the delay in issuing die JASR revisions be- 
cause of the prolonged negotiations between 
DOL and the Office of Management and 
Budget (OME). 

However, two reporting practices reduced 
the comparability of the performance 
measures. First, SDAs typically delayed en- 
rollment of participants until after an orienta- 
tion period so that individuals who dropped 
out early were not enrolled. This pre-enroU- 
ment period usually lasted 3 to 5 days, but in 
some SDAs went on for several weeks. We 
found that SDAs with longer pre-enroUment 
periods have significantiy higher measured- 
entered-employment rates. These results sug- 
gest that some monitoring of this process at 
the State or Federal level would increase the 
comparability of measured performance. 



ERIC 



Second, as discussed above, the fact tiiat 
many SDAs leverage JTPA funds with other 
resources substantially reduces the com- 
parability of the cost standards. The problem 
cannot be rectified by reporting requirements 
because SDAs would find it very difficult to 
estimate the value of other resources (for 
example, the State subsidy to community 
colleges). 

State Policies 

State Performance Standards. States 
are increasingly adopting State performance 
standards to further their goals for the JTPA 
system. The study demonstrated that States 
can affect the direction of the JTPA program 
by implementing State performance stand- 
ards to supplement the Federal standards, in 
particulrr. State standards for service to tar- 
get groups furthered tiie State's client 
priorities for the JTPA program. The success 
of these policies should encourage other 
States to attempt to play a leadership role if 
they desire to do so. 

Additional Adjustment Procedures for 
the Federal Standards. Although tiie effects 
are not entirely consistent, the qi . 'tative 
results suggest that State policies ' ■ ' r specify 
procedures to obtain additional adjustments 
beyond the model do increase service to 
some hard-to-serve groups and encourage 
provision of basic skills remediation to 
youth. However, in our case studies, we 
found that both State and SDA staff were not 
particularly comfortable with the adjustment 
process as it now exists. State staff felt un- 
sure about how to establish equitable criteria 
for adjustments beyond the model and about 
how to determine the appropriate size of ad- 
justments. SDA staff often felt that they did 
not understand the statistical basis of the 
models well enough to justify adjustments. 
In addition, some staff felt that requesting ad- 
justments was admitting failure and, par- 
ticularly when SDAs competed for funds, 
that it was not fair to "go begging." 
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One SDA director suggested an alterna- 
tive process tiiat he felt would be more ac- 
ceptable. Specifically, he suggested that 
SDAs be allowed to request waivers from the 
performance standards for projects that met 
well-specified criteria, for example, for start- 
ing up a prograni for high-risk clients. This 
would be more acceptable, he said, because 
the standards for the remaining projects 
would not be affected, so that all SDAs 
would still be abiding by the same rules. 

Hard-to-Serve Policies. State policies to 
encourage service to hard-to-serve groups in- 
clude additional State performance standards, 
integrating service to hard-to-serve groups - 
into the calculation of incentive payments, 
and identifying priority groups. The evalua- 
tion indicates that States can play an effec- 
tive leadership role in focusing JTPA 
programs on the needs of particular hard-to- 
serve groups through a variety of incentive 
awards adjustments, in addition to the im- 
plementation of special State standards. 

Incentive Policies. The JTPA legislation 
requires that incentive payments be awarded 
based on the extent to which standards are ex- 
ceeded. States policies vary widely in how 
that requirement is implemented. Policies 
that place a strong emphasis on exceeding 
standards are found to lead SDAs to reduce 
service to hard-to-serve groups, reduce 
provision of basic skills remediation, and in- 
crease the employment focus of both the 
adult and youth programs. These are largely 
unintended effects of both the legislative re- 
quirement and of the specific State policies. 
The results of this evaluation strongly sug- 
gest that these policies should be 
re-examined. 

Another important aspect of incentive 
policies is the weight placed on each stand- 
ard. High weights on the cost standard were 
found to reduce service to^welfare recipients 
and dropouts. States that equally weight the 
percentage by which each standard is ex- 
ceeded implicitiy give substantially greater 



weight to the cost standards because costs are 
more variable among SDAs. These policies 
should also re-examined 

The case studies also indicate that incen- 
tive payments are very important to SDAs be- 
cause of two other Federal policies. First, the 
allocation formula bases funding on the un- 
employment rates. Incentive payments help 
SDAs cushion the decline in JTPA funding 
brought on by an improving economy. 
Second, many SDAs are squeezed by the 
15% limit on administrative costs. Because 
30% of incentive funds can be used for ad- 
ministrative costs, these funds are particular- 
ly valuable to many SDAs. 

Sanction Policies, Many States have not 
developed specific policies for sanctioning 
SDAs for poor performance, and even when 
these policies are in place. States are reluc- 
tant to enforce them. Tnc threat of reorganiza- 
tion has little effect on SDA beha-ior. 

Technical Assistance. The study find- 
ings suggest that SDAs would benefit from 
additional State technical assistance to clarify 
the State's intention in designing its perfor- 
mance-standards policies and to discuss how 
the performance standards should interact 
with other client and service priorities in fur- 
thering JTPA program priorities. 

Conclusions 

1. The performance standards need to be 
balanced by local client and service 
goals to provide a useful guide to 
managing JTPA programs. If taken as 
the sole statement of program goals, 
the standards can lead to unintended 
effects of reducing service to hard-to* 
serve groups and decreasing the inten- 
sity of JTPA services. 
The study found that States, SDAs, PICs, 
and many service providers express strong 
support for the idea of cost-effectiveness and 
accountability for outcomes as an important 
part of the JTPA program. When SDA staff 
talk about the effect that performance stand- 
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aids have had on the design and operation of 
their programs, they generally do not ques- 
tion the premise that they should be account- 
able for program outcomes. 

The idea of performance accountability, 
however, has been embraced by some in- 
dividuals in the JTPA system to the extent 
that performance goals sometimes receive 
more emphasis than other program objec- 
tives, such as client priorities and goals about 
what types of cervices to offer. The absence 
of clear client and service goals in some 
States and SDAs is unfortunate, because it 
can result in performance goals receiving 
undue emphasis. Performance standards had 
notable unintended effects of reducing ser- 
vice to the hard to serve and decreasing the 
intensity of services only in SDAs without 
well-defined client or service goals. 

In general, however, the performance 
standards are not preventing SDAs from pur- 
suLig their local client and service goals and 
are not dramatically altering the types of 
clients served by JTPA programs or the types 
of services being offered. 
2. Although the performance^standards 
system as a whole appears to be work** 
ing^ the cost standards are more 
problematic than the other standards. 
Reducing the emphasis on the cost 
standards would reduce unintended im- 
pacts of the standards. 
The Federal cost standards have the most 
unintended effects and are the least com- 
parably measured of all the Federal perfor- 
mance measures. In addition, SDAs in States 
with incentive policies that emphasize perfor- 
mance in excess of the standarr' often find 
that reducing costs is the easiest and most 
dramatic way to perform beyond their stand- 
ards. 

Thus, concerns about exceeding the cost 
standards tend to reduce the average intensity 
of services offered by some SDAs. These 
findings suggest that alternatives that reduce 
the emphasis on the cost standards should be 



explored. 

3. State incentive policies have unin- 
tended effects and need to be 
re*examined* States should be assisted 
in developing policii^s that enable them 
to fulfill their potential of providing 
leadership to the JTPA system. 

The State incentive policies that en- 
courage SDAs to perform as high as possible 
on all the standards have unintended effects. 
The quantitative analysis showed that these 
policies have modest but consistent effects of 
reducing service to hard-to-serve clients and 
reducing the provision of expensive program 
services, such a basic skills remediation. 

Because the e impacts are probably not 
intended. States should carefully review their 
performance-standards policies. One solution 
may be for States to establish more explicitiy 
client priorities in their performance-stand- 
ards policies. Another solution may be for 
States to develop policies that are more 
neutral in their effect on SDA client targeting 
decisions and service design choices. 

Solutions should be explored to permit 
States to reward SDA performance that is 
"exemplary" rather than "average" while 
avoiding unintended effects on clients and 
services. 

4. The youth standards are currently in- 
fluencing SDA decisions about client 
targeting and program design. One 
change that would reduce unintended 
^'^^ndards impacts would be to estab- 
lish separate standards for in-school 
and out-of-school youth. 

There is intense debate within and among 
SDAs about whether youth programs should 
be oriented toward immediate employment or 
toward employment-competency outcomes. 
We saw no signs of an emerging consensus 
on this issue. 

The performance-standards system re- 
quires SDAs to report both employment and 
cmployment-compett :y outcomes for all 
youtii participatiiig in die JTPA program. 
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This compromise, however, has not been 
completely satisfactory from the SDAs' 
perspective. SDAs that emphasized services 
to in-school youth often found it difficult to 
meet the youth entered-employment-rate 
standard, while SDAs that emphasized 
employment-oriented services to youth who 
have left school often found it difficult to 
meet the youth positive-termination-rate 
standard 



Although most SDAs have developed 
program designs that enable them to meet the 
youth performance standards, there is little 
enthusiasm and considerable confusion at the 
SDA level over the youth standards. One 
change that might increase local discretion 
would be to establish separate standards for 
in-school youth and out-of-school youth. 
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Recommendations of the Commission 



The National Commission for Employ- 
ment Policy believes that this report is an ex- 
tremely valuable contribution to research on 
the Job Training Partnership Act (JTPA), and 
one whose findings have implications that go 
well beyond the scope of this project — to 
determine the Lnpact of performance stand- 
ards on clients, services, and costs. Insights 
gained from the comprehensive analysis of 
both quantitative and qualitative data call for 
actions from national. State, and local 
policymakers that are not limited to perfor- 
mance standards. Therefore, the 
Commission's recommendations include 
both general ones as well as specific recom- 
mendations concerning changes to the perfor- 
mance management system itself. 

General Recommendations 
National Policy 

The Act balances legislative and Federal 
objectives for the program with local discre- 
tion designed to allow the program to meet 
local needs. The Commission believes that 
targeting of JTPA programs works very well 



but that the Congress must be very clear 
about who the Act is intended to serve while 
maintaining the balance between national ob- 
jectives and local discretion. Further clarity 
may be needed in defining the three major tar- 
get groups mentioned in the Act — welfare 
recipients, dropouts, and youth. For example, 
with respect to welfare recipients, since legis- 
lation affecting both the welfare system and 
JTPA^s service to welfare recipients has been 
under consideration by the Congress, it may 
be timely to clarify that long-term welfare 
recipients (those receiving welfare for 2 years 
or longer) are the welfare recipients best 
servc^ In JTPA programs. In the area of 
dropouts, it may be necessary to define 
dropouts as youth (since the lack of a high 
school credential, for a youth more so than 
for adults with work experience, is a sig- 
nificant barrier to employment). For all 
dropouts, basic skills deficiencies appear to 
be more of a barrier to full workforce par- 
ticipation than educational status. 

Probably more than any other vehicle, 
performance standards have been used to 
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transmit national policy in TTT x programs. 
Sometimes these signals, transmitted through 
the choice of performance measures and na- 
tional standards, have not always been clear 
to State and local program managers. Recent- 
ly, much progress has been made in using 
performance standards to reinforce, ahready 
stated policy goals and objectives. 

However, we believe that performance 
standards should reinforce policy, not estab- 
lish policy goals. The Commission recom- 
menids that the Department of Labor clearly 
articulate (1) policy objectives for the 
program apart from performance standards, 
(2) specific policy on the provision of basic 
skills for both youth and adults, (3) objec- 
tives about the intensity of training (and the 
type of training appropriate for JTPA par- 
ticipants) that are desirable, (4) appropriate 
outcomes for both in-school and out-of- 
school youth, and (5) objectives concerning 
the State role in promoting such objectives. 

The Department of Labor needs to give 
additional attention to the collection of and 
definition of data elements to support policy 
objectives and work with the Office of 
Management and Budget to implement a 
more carefully defined management informa- 
tion system. 

State Policy 

This report has provided new insight into 
the role of States in the perforaiance manage- 
ment system. As the report indicates. State 
policies can either promote or dissuade PICs 
and SDAs from providing intensive service 
to the hard to serve. The Con: ^lission 
believes that better use can hz made of these 
policy tools through the goals and objectives 
detailed in the Governor's Coordination and 
Special Services Plan, through incentive 
award policies for exceeding performance 
standards, through sanctions for failure to 
meet performance standards, and through the 
use of incentives ior serving the hard to serve 
as specified in Section 202 (b)(3)(B) of the 
Act. The Commission recommends that 



States improve their incentive policies to 
promote the provision of service to those 
clients deemed to be most in need. One par- 
ticular mechanism to achieve a clearer focus 
on the hard to serve is through better use of 
incentives for serving the hard to serve. 

Local Policy 

The Commission is concerned that one 
p.artner in the Act, the service provider, is 
still largely overlooked in the performance 
management system. Since JTPA services 
are often provided by entities other than the 
administrative entity, we are concerned that 
contractors need to be included in both the 
risks and the rewards of programs. The Com- 
mission recommends that PICs develop 
reward sy^ ems built into contracts for ser- 
vice providers who are successful in provid- 
ing intensive service to the most in need in 
the JTPA population, particularly for welfare 
recipients and dropouts, as well as other 
groups the PLCs target for service. 

Specific Performance Management 
Recommendations 

The Commission calls attention to the 
fact that all recommendations made here are 
provided based on the performance manage* 
ment system in place during PY 86-87 
without the benefit of analyzing the effect of 
the implementation of post-program perfor- 
mance measures and standards. This report 
notes a fair degree of support for measuring 
the employment and earnings of individuals 
after they leave JTPA programs, but since no 
standards were in place during the year of 
this study, we could not directly assess their 
effect on who is served, the type of service, 
or the cost of such service. 

We do believe, however, that most of the 
recommendations offered here will apply 
once post-program standards have been fully 
implemented into the JTPA system. 

Choice of Measures 
1. The Commission strongly supports the 
concept of cost-effectiveness inherent in per- 
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formance standards and recognized in the 
Act but believes that too much emphasis on 
efficiency measures (in this case two perfor- 
mance standards deiling with cost) may have 
had the unintended effect of precluding the at- 
tainment of effective programs to meet the 
goals of the Act. We are also concerned 
about the comparability of the cost informa- 
tion across programs. JTPA programs have 
made great strides in leveraging the resources 
of other systems to serve the economically 
disadvantaged; however, this may mtrjx) that 
the actual cost of serving some clients is 
undeneported in some programs. 

Therefore, the Commission recommends 
that Section 106 (b)(4), which requires the 
Secretary to prescribe performance stand- 
ards relating gross program expenditures to 
various performance measures, be amended 
to direct that cost-efficiency be monitored by 
States. 

2. The Commission endorses the concept 
that employment is the major objective of the 
Act and believes that the Act wisely em- 
phasizes increases in employment and earn- 
ings as major goals. However, the Act also 
recognizes the importance of outcomes other 
than employment for youth — namely, the 
attainment of youtli employment competen- 
cies; completion of major levels of school; 
and enrollmtjnt in other training programs. 

As the report's findings indicate, one of 
the intended effects of performance standards 
for youth that was realized was the develop- 
ment of programs to enhance a young 
person's future employability through the at- 
tainment of youth employment competencies. 
While the Commission favors this approach 
to youth programs, we are concerned about 
the continuing problems in the youth 
measures. The Commission endorses preserv- 
ing local programs' discretion about the rela- 
tive emphasis of serving in-school and 
out-of-school youth. The new youth measure 
introduced for PY 88 will not fully address 
the differing goals of in-school programs that 
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focus on improving basic skills competencies 
and dropout prevention strategies and out-of- 
school programs that should focus on 
employability development (particularly in 
the area of basic skills) for youth for whom 
the final outcome is employment. The Com- 
mission therefore recommends that the 
Department of Labor develop separate 
reporting for in-school and out-of-school 
youth programs and develop appropriate 
outcomes for both. 

Exceeding Perforniance Standards 
The evaluation of the effect of perfor- 
mance standards also points to another area 
of concern in the legislation that we believe 
requires remedy. Section 202 (b)(3)(B) of the 
Act directs the Governor to provide incentive 
awards for programs "exceeding perfor- 
mance standards." As indicated in this evalua- 
tion, this language may have had the 
unintended effect of promoting overperfor- 
mance and setting up competition among 
SDAs for incentive funds at the expense of 
some clients and services that might other- 
wise be offered for the hard to serve. One of 
the original principles upon which the perfor- 
mance management system was based is that 
SDAs should be judged against their own 
local circumstances, that is, economic condi- 
tions, characteristics of the population to be 
served, and so forth. The Commission recom- 
mends that the Congress clarify its intent to 
promote service to the hard to serve by 
changing Section 202 (b)(3)(B) of the Act to 
provide for incentive awards based on ''meet- 
ing" performance standards. At a minimum. 
States should include provisions in their in- 
centive policies that do not promote overper- 
formance at the expense of service to the 
hard to serve. 

The Commission believes that further 
discussion is needed about more specific 
guidance to the States concerning "incentives 
for serving the hard to serve." Therefore, the 
Commission recommends that an amendment 
to Section 202 (b)(3)(B) be considered to re- 
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quire States to provide such incentives. 

Encouraging the Provision 
of Basic Skills 

We are particularly concerned about en- 
couraging the attainment of basic educational 
skills for both youth and adults. We have sup- 
ported the efforts of the Secretary in em- 
phasizing the role of tra^ining programs like 
JTPA in ensuring that basic skills remedia- 
tion is a significant part of any employability 
program. The Commission is also interested 
in ways to provide incentives to JTPA 
program operators to link improvements in 
basic skills with occupational training and ul- 
timately employment. One way to improve 
the performance management system's 
ability to do so may be to collect data that 
could be used to adjust standards or provide 
bonuses for programs that emphasize the 
provision of basic skills in addition to 
employment. The Commission therefore 
recommends that the JTPA Annual Status 
Report be changed to Include information for 
both adults and youth who, in addition to 
entering employment, attain basic skills 
while enrolled in JTPA progams (whether 
or not attained with JTPA funds). 

Adjustment to Standards 

The Commission is pleased that the ad- 
justment models used by the majority of 
States to establish local SDA standards have 
the intended effect of promoting service to 
some target groups for which employment 
may be more difficult to achieve (the so- 
called hard to serve); however, we are con- 
cerned that the use of any further adjustments 
does not seem to be an effective approach for 
a variety of reasons. JTPA programs must en- 
courage innovative strategies to solve the dif- 
ficult problems faced by those served under 
the Act. While we commend the Department 
of Labor's effort to promote additional adjust- 
ments, the Commission believes that it is 
time to consider alternative approaches. 

We do not believe that enough work has 



been done in this area to suggest specific ap- 
proaches, but some suggestions from recent 
Commission publications and from the work 
of other groups should be explored. These 
studies suggest utilizing incentive approach- 
es rather than (or in addition to) the "hold 
harmless" approach currently used in the ad- 
justment models. Some of the specific op- 
tions that should be considered include the 
use of incentive funds for serving at-risk 
clients based on pre-enrollment criteria such 
as targeting individuals with limited prior 
work experience, with basic skills deficien- 
cies, and who are long-term welfare recip- 
ients; and the use of waivers to performance 
standards to encourage innovative strategies 
for programs that serve the most at risk, A 
waiver could also be used for programs ex- 
pected to transcend program years, thus 
promoting longer term programs when 
suitable for the hard to serve. The Commis- 
sion recommends that the Department of 
Labor explore such alternative approaches 
which may go ''beyond the adjustment 
models'' to promote positive incentives to 
serving individuals most at risk of future 
employment problems. 

Incentive Awards 

The Commission is concerned that some 
incentive award policies have become so 
complex that they may be having the unin- 
tended effect of dissuading service from the 
hard to serve simply because the policies are 
difficult to understand. This report has noted 
a number of both positive and negative ap- 
proaches currently in use by States that affect 
who is served, the type of service, and the 
cost of service. Translation of goals and ob- 
jectives through the use of the incentive 
award rysteni needs to be kept as simple as 
possible to have the desired effect on local 
programs. The Commission therefore recom- 
mends that technical assistance be provided 
to States to ensure that incentive policies 
have the effect States intended and that 
States clearly reward service to clients most 
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in need. 

Technical Assistance 

The Commission is p'.eased that, for the 
most part, PIOs and SD As have a fairly good 
understanding of the performance standards 
system, but we are concerned that a different 
strategy in the delivery of technical assis- 
tance is needed. The Department of Labor is 
tc be commended for providing assistance on 
performance standards, including their 
development and ways to adjust standards. 
However, we are concerned that their techni- 
cal assistance on standards does not suffi- 
ciently promote the policy objectives impor- 



tant to system-wide program improvement 
aimed at better preparation of the workforce. 
The Commission recommends that future 
technical assistance efforts be focused on im- 
proving programs and developing innovative 
strategies targeted to long-term employment 
of the most in need. Performance standards 
should be included as one aspect of how to 
plan and manage programs. Such a strategy 
should include PICs and service providers 
and must emphasize how to integrate 
program goals about who to serve and the 
types of service with performance expecta- 
tions into contracts. 
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